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Spike-timing-dependent plasticity (STDP) has recently been shown in some physiological studies. 
STDP depends on the precise temporal relationship of pre- and post-synaptic spikes. Many authors 
have indicated that a precise balance between long-term potentiation (LTP) and long-term depres- 
sion (LTD) of STDP is significant for a stable learning. However, a situation in which the balance is 
maintained precisely is inconceivable in the brain. Using a method of the statistical neurodynamics, 
we show robust retrieval properties of spatio-temporal patterns in an associative memory model 
against the imbalance between LTP and LTD. When the fluctuation of LTD is assumed to obey a 
Gaussian distribution with mean and variance S 2 , the storage capacity takes a finite value even at 
large 8. This means that the balance between LTP and LTD of STDP need not to be maintained 
precisely, but must be maintained on average. Furthermore, we found that a basin of attraction 
becomes smaller as 8 increases while an initial critical overlap remains unchanged. 

PACS numbers: 87.18.Sn, 89.70.+C, 05.90.+m 



I. INTRODUCTION 

Recent experimental finding indicates that synaptic 
modification in cortical neurons depends on the precise 
temporal relationship between pre- and post-synaptic 
spikes 0, 0, 13- In particular, pre-synaptic spikes that 
precede post-synaptic firing induce long-term potentia- 
tion (LTP) by no more than 20 ms, while those that 
follow post-synaptic firing induce long-term depression 
(LTD), with a rapid transition (a few ms). The magni- 
tude of synaptic modification decays exponentially with 
the time interval between pre- and post-synaptic spikes. 
This form of synaptic modification has been called spike- 
timing-dependent plasticity (STDP) 01 or temporally 
asymmetric Hebbian learning (TAH) 

The functional role of STDP has been investigated by 
many authors. They showed that STDP is a mechanism 
for synaptic competition 0, IE IE Bi IE I pl or a learning 
mechanism of sequential patterns |lC)UlU I rlL HIH IlH ITa 
Il6j . Asymmetric learning window depending on spike 
timing like STDP has been studied and shown to be ap- 
propriate learning rule for sequential patterns |l7lll8llT^ . 
However, this asymmetric learning rule does not involve 
LTD. Some authors showed that the balance between 
LTP and LTD of STDP is significant for a stable learn- 
ing 0, 0, 0, 0. In our previous work, we analyti- 
cally showed that STDP has the same qualitative effect 
as the covariance rule when the spatio-temporal patterns 
are stored since the differences between spike times that 
induce LTP or LTD are capable of canceling out the ef- 
fect of the firing rate information In the brain, a 



situation in which the balance is maintained precisely is 
inconceivable. The data points obtained by experiments 
are fluctuated in the different trials [lj,y|- Therefore, it 
is meaningful to discuss more biological plausible situa- 
tion to investigate the neuronal mechanism for sequential 
learning in the brain. Some authors numerically inves- 
tigated the impact of the imbalance between LTP and 
LTD on the network properties 0, 0] . 

The aim of this paper is to analytically discuss the 
retrieval properties of spatio-temporal patterns in an as- 
sociative memory model that incorporates the imbalance 
between LTP and LTD of STDP using a method of the 
statistical neurodynamics 0, |2£j, EI ■ According to our 
previous work, when the balance is not precisely main- 
tained, it is impossible to cancel out the information of 
firing rate. Consequently, a cross-talk noise diverges. 
However, if the magnitudes of LTP and LTD are equiva- 
lent on average in a learning process, it may be possible 
to stably retrieve spatio-temporal patterns. Since the ra- 
tio of LTP and LTD is crucial, the magnitude of LTD 
changes while that of LTP is fixed. We found that the 
storage capacity takes a finite value even at large S when 
the fluctuation of LTD is assumed to obey a Gaussian dis- 
tribution with mean and variance S 2 . This implies that 
the balance between LTP and LTD of STDP need not 
to be maintained precisely, but must be maintained on 
average. This mechanism might work in the brain. Fur- 
thermore, we found that a basin of attraction becomes 
smaller as S increases while an initial critical overlap re- 
mains unchanged. 



II. MODEL 
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The model contains N binary neurons with recipro- 
cal connections. Each neuron has a binary state {0, 1}. 



2 



We define discrete time steps and the following rule for 
synchronous updating: 



N 

Ui(t) = y^jjjXj(t), 

3=1 

Xi{t + 1) = F[ Ui {t)-9), 

1. u > 
0. u < 0, 



F(u) 



(1) 

(2) 
(3) 



where Xi(t) is the state of the i-th neuron at time t, Ui(t) 
is the internal potential of that neuron, and 8 is a uni- 
form threshold. If the i-th neuron fires at t, its state is 
Xi(t) = 1; otherwise, Xi{t) = 0. Jy is the synaptic weight 
from the j-th neuron to the i-th neuron. Each element 
j£ of the /i-th memory pattern £^ = (£^,£21 
generated independently by 



Prob[£f = 1] = 1 - Prob[£f = 0] = /. 



(4) 



The expectation of £ M is E[£^] = /, and thus / is consid- 
ered to be the mean firing rate of the memory pattern. 
The memory pattern is sparse when / — ► 0, and this 
coding scheme is called sparse coding. 

The synaptic weight Jij follows the form of synaptic 
plasticity, which depends on the difference in spike times 
between the i-th (post-) and j-th (pre-) neurons. The 
time difference determines whether LTP or LTD is in- 
duced. This type of learning rule is called spike-timing- 
dependent plasticity (STDP). The biological experimen- 
tal findings show that LTP or LTD is induced when the 
difference in the pre- and post-synaptic spike times falls 
within about 20 ms |3|. We define a single time step in 
equations (^0) as 20 ms, and durations within 20 ms are 
ignored. The learning rule based on STDP conforms to 
this equation: 



one time step after the i-th neuron, £f _ = $ = 1. Since 
the ratio of LTP and LTD is crucial, the magnitude of 
LTD changes while the magnitude of LTP and the time 
duration are fixed, is generated independently and 
obeys a Gaussian distribution with mean e and variance 
5 2 : ~ Af(e,S 2 ). Fig shows the time function of 
STDP in our model. When = 0, the balance between 
LTP and LTD is precisely maintained and then the model 
is equivalent to the previous model fl6| . A sequence of p 
memory patterns is stored by STDP: — * £ 2 — > • • • — > 
£ p — ► ^ 1 — * • • •. In other words, is retrieved at t = 1, 
£ 2 is retrieved at t = 2, and is retrieved at t = p + 1. 
There is a critical value ac of the loading rate, so that the 
loading rate larger than ac makes retrieval of the pattern 
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FIG. 1: The time function of STDP in our model. LTP is 
induced when the j-th neuron fires one time step before the 
i-th one. LTD is induced when the j-th neuron fires one time 
step after the i-th one. follows a Gaussian distribution 
with mean e and variance <5 2 . 
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The number of memory patterns is p = aN, where a is 
defined as a loading rate. LTP is induced when the j-th 
neuron fires one time step before the i-th neuron, +1 = 
= 1, while LTD is induced when the j-th neuron fires 



III. THEORY 

In this section, we derive dynamical equations that de- 
scribe the retrieval properties of the network. In this 
paper, we consider the thermodynamic limit: N — ► oo. 
The i-th neuronal internal potential Uj(t) at time t can 
be expressed as 



N N p 

,(*) = E jm® - Nm _ f) E E {ff+'ff - ft + Off 'ff} «*(*) (8) 

3=1 J V J ' 3=1 A*=l 



N v * N p 



EE(^ - er x $)*i (*) - ivvtt-tt E E 4 '^w. (7) 



Using the periodic boundary condition of £f +1 = £?■ and with £f = — f. Using this relationship, Ui(t) 

CO _ rf> V^P (eV-+ 1 <:H _ £•/*-! _ V^P (FH+lFV _ 
Si - Si ! Z^u=lVSi Sj Si Sj ; — Z^u=llSi Sj 
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is given by 



N p 



Ui(t) 



J=l P 
JV p 



r— nEEC^r'^-w (8) 



E^-erv^) 



1 JV p 

-^vfTwyEEC^r 1 ^-^) o) 

JV p 



1 



EE-' (io) 



where m^ft) is an overlap between £ M (i) and a:(£) and is 
given by 



»"(*) = 



1 



A" 



Nf(l-f) 



(11) 



The first term in equation (|10|l is a signal term for the 
retrieval of the target pattern The second and 

third terms are a cross-talk noise term that represents 
contributions from non-target patterns other than 
and that prevents from being retrieved. The third 
term is also a compensation term originated by a devia- 
tion from the balance between LTP and LTD of STDP. 
Since this term is order of N with respect to N, O(N), 
when e 0, it diverges in the thermodynamic N limit: 
N — > oo. This means that the stored limit cycle using 
the present learning rule (equation (JSJ) is unstable in the 
limit of N — > oo when e ^ 0. Therefore, we will discuss 
the e = case: e£ ~ Af(0,S 2 ). 

We derive the dynamical equations using the method 
of the statistical neurodynamics 1 2(1 121| . When it is 
possible to store a pattern sequence, a cross-talk noise 
term, that is, the second and third terms in equation 
(|TU|> is assumed to obey a Gaussian distribution with 
the average and time-dependent variance a 2 (t) [Tril20| . 
We derive the recursive equations for m'(i) and <J 2 (t) to 
investigate whether the memory pattern £' is retrieved 
or not. Since m'(£) depends on <r 2 (t), we derive cr 2 (t). 
The dynamical equations are derived as 



1-2/ 



erf (0o ) 



^erf(^) + 4rf(</> 2 ), (12) 



m\t) 

ff2 w = E^+^+wi-^n^-k+i) 



6=1 



aS 2 



(l-/) : 



:?(*), 



(13) 



[/(*) 



2^^ (1 " 2/ + 2/ 



+/(l-/)(e-^+e-^)}, (14) 
(*) = i{l-(l-2/ + 2/ 2 )erf(0 o )-/(l-/)(crf(0i) 
+erf(0 2 ))}. (15) 



where erf(y) = -2- exp (—u 2 )du, and 



V2(T(t-l) ' 



_ -m t ~ 1 (t-l)+8 



m*~Vt-l)+g 



i ;-t-p p a: 

^ - ^2a{t-l) ' ^ ~ v^trCt-l) ' hVja ~ a\{b- a y. 
and a! is the factorial with positive integer a. The de- 
tail derivation of the dynamical equations is shown in 
Appendix A. 



IV. RESULTS 

FigElshows the dependence of the overlap m'(i) on the 
loading rate a when the mean firing rate of the mem- 
ory pattern is / = 0.1, and the threshold is 8 = 0.52, 
which is optimized to maximize the storage capacity, (a) 
is the case at S = 0.0, (b) is at 8 = 1.0, and (c) is at 
S = 2.0. The lines show analytical results obtained by 
the dynamical equations (|12II15|) . The upper line denotes 
the steady-state values of the overlap m'(£) in retrieval 
of the pattern sequence, m (t) is obtained by setting 
the initial state of the network at the first memory pat- 
tern: x(l) = £ 1 . Setting the initial values at m 1 (l) = 1, 
cr 2 (l) = 2a f + tjz^t, U(l) = and q(l) = / and us- 
ing the dynamical equations (|12H15|I . m*(t) is obtained. 
When the overlap at the steady state is smaller than 0.5, 
the critical loading rate a is regarded as the storage ca- 
pacity etc- The storage capacity ac is 0.27(a), 0.178(b), 
and 0.087(c). The lower line indicates the dependence of 
an initial critical overlap mc on a. The stored pattern 
sequence is retrievable when the initial overlap m 1 (l) is 
greater than the critical value mc- The region in which 
m 1 (1) is larger than mc represents the basin of attraction 
for the retrieval of the pattern sequence, mc is obtained 
by setting the initial state of the network at £ : with ad- 
ditional noise. We employ the following method to add 
noise. 100s% of the minority components (xi(l) = 1) are 
flipped, while the same number of majority components 
(xi(l) = 0) are also nipped. The initial overlap m x (l) is 
given as 1 — -^j . Then the mean firing rate of the net- 
work is kept equal to that of the memory pattern, /. The 
other initial values are equivalent to the upper line case. 
When the overlap at the steady state is smaller than 0.5, 
the initial overlap m 1 (l) is regarded as the initial criti- 
cal overlap mc- The data points and error bars indicate 
the results of computer simulations of 11 trials with 5000 
neurons: N = 5000. The results are obtained from the 
equations (Q-JSJ). The data points indicate median values 
and both ends of the error bars does 1/4 and 3/4 devia- 
tions. A discrepancy between the values of mc obtained 
by the computer simulations and the analytical results 
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FIG. 2: The dependence of ac and the basin of attraction 
on the loading rate at / = 0.1 and 6 = 0.52. The lower 
line represents the initial critical overlap and the upper line 
does the overlap at the steady state. The data points and 
the error bars show the results of computer simulation of 11 
trials at N = 5000. (a):<5 = 0.0, (b):<5 = 1.0, (c):<5 = 2.0. 
a c = 0.27(a), 0.178(b), 0.087(c). The basin of attraction 
decreases as 8 increases. 



is originated from the finite size effect of the computer 
simulations |l6[ . 

Fig|2|shows the dependence of the storage capacity ac 
on the standard derivation 8 of the fluctuation of LTD at 
/ = 0.1 and 9 = 0.52. The solid line shows the analytical 
results obtained by the same procedure to obtain ac in 




FIG. 3: The dependence of the storage capacity ac on 8. 
The solid line shows the analytical results. The data points 
and error bars show the results of computer simulation of 10 
trials at N = 5000. Both results are obtained at / = 0.1 and 
8 = 0.52. As the variance 8 2 of the fluctuation increases, ac 
decreases. 



Fig0 The data points and error bars show the results of 
computer simulation of 10 trials at N = 5000. The means 
and standard deviations of ac of 10 trials are plotted as 
the data points and the error bars, respectively. As the 
variance S 2 increases, ac decreases. In other words, the 
model is robust against the imbalance between LTP and 
LTD of STDP. Thus, the balance does not need to be 
maintained precisely, but must simply be maintained on 
average. 

Fig 01 shows an asymptotic property of ac in a large 
limit of S 2 . The solid line shows the analytical results 
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FIG. 4: An asymptotic property of ac in a large limit of 8 2 
at / = 0.1 and 9 — 0.52. The solid line shows the analytical 
results while the dashed line shows log 10 ac = log 10 -§s — 
1.13. ac converges to as order of with respect to 8, 
0{ji), in the large 8 2 limit. 



obtained by the same procedure to obtain ac in FigEl 
at / = 0.1 and 9 — 0.52 while the dashed line shows 
log 10 ac = log 10 -ly — 1.13. This figure indicates that ac 
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converges to as order of ^ with respect to 8, O(^), in 
the large S 2 limit ^ 

Next, we discuss the dependency of the basin of attrac- 
tion on 6. Each region between the upper line and the 
lower line in Fig. Efa),(b),(c) shows the basin of attrac- 
tion at S = 0.0, 1.0, 2.0, respectively. The basin becomes 
smaller as a value of S increases. However, the initial crit- 
ical overlap mc is unchanged. To introduce a threshold 
control scheme is known to enlarge the basin of attraction 

ElSollII. 



V. DISCUSSION 

In this paper, we investigated the impact of the im- 
balance between LTP and LTD of STDP on the retrieval 
properties of spatio-temporal patterns, employing an as- 
sociative memory network. We analytically investigated 
the retrieval properties using the statistical neurodynam- 
ics. When the fluctuation of LTD is assumed to obey the 
Gaussian distribution with mean and variance <5 2 , the 
storage capacity takes a finite value even at large 5. This 
implies that the balance does not need to be maintained 
precisely, but must be maintained on average. This mech- 
anism might work in the brain. Furthermore, the storage 
capacity converges to as order O(p-) in the large S 2 
limit. Finally, we found that the basin of attraction be- 
comes smaller as the fluctuation of LTD increases while 
the initial critical overlap remains unchanged. 

We found that the storage capacity takes a finite value 
even at large 5. When 5 is larger than 1.0, LTD some- 
times disappears in the learning process. The spatio- 
temporal patterns seem not to be retrievable. Surpris- 
ingly, even in this situation, the patterns are retrievable. 
This implies that the present model achieves strong ro- 
bustness against the imbalance between LTP and LTD. 



APPENDIX A: DERIVATION OF DYNAMICAL 
EQUATIONS BY THE STATISTICAL 
NEURODYNAMICS 

The detail derivation of the dynamical equations (|12h 
I15J1 is given in this appendix. At first, we give a sketch 
of the derivation. The main point in this derivation is to 
divide an internal potential u.i(t) into two parts, a signal 
term for a retrieval of a target pattern and a cross-talk 
noise term that represents contributions from non-target 
patterns and prevents the target pattern from being re- 
trieved. We evaluate the cross-talk noise term. Specifi- 
cally, the internal potential ut (t) of z-th neuron at time t 
is expressed as (see equation (El) 



m = E^-ervw 



11=1 



N p 



^EEc^r 1 ^*) (ai) 



- r + e«T +1 - er 

N p 



1 



(A3) 



where £f = £f — /, m M (i) is the overlap between £ M (i) 
and x(t) and is given by 



N 



Nf(l - f) ^ 



£#**(*). 



(A4) 



and 



■•(*) - £(# +1 -3'"V , (*) 

N p 



1 



EE':; : C l :;',v/;.;A5i 



The first term in equation (|A3|) is the signal term and 
the second term is the cross-talk noise term. Since Xi (t) 
in equation i|A4(l depends on £f , the distribution of the 
cross-talk noise term Zi(t) is unknown. The dependence 
on is extracted from Xi (t) using the Taylor expansion 
(see equation (IA10|l L In the thermodynamic limit, N — > 

oo, m^it) tends to be deterministic. Therefore, x\^ (t), 
which denotes that it does not include is independent 
of . This enables us to assume that the cross-talk noise 
term Zi(t) obeys a Gaussian distribution with mean and 
variance a 2 [T(| ■ Since the distribution of the cross- 
talk noise term is known, the recursive equation of the 
overlap is obtained (see equation ^35j). 

To extract the dependence on £f from Xi(t), the state 
of the z-th neuron at time t + 1 is transformed: 

Xi(t + 1) = F{ Ui {t)-6) 



i 



N p 



Nf(l-f) 



EEi^w-" 



3 = 1 v=l 



(A6) 

The first term in the function F(-) of equation i|A6[) is 
transformed using the periodic boundary condition of 
^=£1 and 

EK? +1 -erV(*) 
= E^+^w-Efr 1 ™^) 

= E^w-E^*) 



6 



L (t)}. 



(A7) 



The second term in the function F(-) of equation (| Af3|> is 
transformed using the periodic boundary condition: 

1 n P 

= ^/(i _ /) E E (as) 

To extract the dependency on £j from + 1), using 
the equation (|A7() and HA8[) . a;(t + 1) is divided into two 
parts, the terms which include ff and the terms which 
do not include 



(A9) 

At time t, the pattern is designed to be retrieved. 
Therefore, we can assume that m (t) is order of 1 with 
respect to N, m*(i) ~ 0(1), and m^(t)(ij, ^ t) is order 
of 1/-/AT with respect to N, m^(t) ~ 0(l/y/N). Since 
m^(t) ~ 0(1/ y/N), m^ _1 (i) and m^ +1 (i) are order of 
with respect to iV. Since m^ +1 (i) - 0(1/ VW) 
and e^- ~ 0(1), the second term in equation (|A9|) can be 

considered to be order of 1/yN with respect to N. In 
the thermodynamic limit, N — > oo, the first and the sec- 
ond terms in equation (|A9|) are small. Using the Taylor 
expansion up to the first order of Xi(t + 1), Xi(t + 1) is 
transformed: 



i Ti(t + l)=a;| /i) (t + l)+uJ w (t)a:; w (t+l) 1 (A10) 



(t+i) = ^(Eerl^- 1 ^)-™^ 1 ^)} 



where x^'(t + 1) is independent of x'^'it + 1) is 
differential of x+(t + 1), and 

u^ } (t) = ^{m^\t)-m^(t)} 
1 N 

- jv/TJ^T) E ^(*).( A1 i) 

zj^t + l) = F(u,(t) - uf^(t) - 0), (A12) 
\t + l) = F'( Ul (t)-u^ } (t)-9). (A13) 



v=l 



N p 



' Nf(i~ f) EE^r^w 

, (er{m"- 1 (t)-m"+ 1 (*)} 

1 w 



JV p 



Nf(l- f) ^ ^ e ^ l %' + 



'(AO , 



We assume that the function F(-) is differentiable. This 
assumption is valid since the average of -F(-) over a Gaus- 
sian noise term will be taken in a later step (see equation 
(IA34(l ). For /j ^ i, the overlap m^(t) is expressed as 



m"(t) 



JV 



Nf(l-f)^ 

JV 



(A14) 



v , ;) E(C) 2 {m"" 1 ^ - 1) - r^ +1 (t - 1)} ^(t) 



JV JV 



■ ( Njthv ) 2 e^e ^ (* - d-: (ai) (*) ■ 



(A15) 



i=l i=i 



If the averages over £j([i ^ t) and £%(^ t) are taken in the right-hand side of equation (|A15fl . the third term 
vanishes since E[e^J = 0. Since the third term including Xj(t— 1) depends on both ijj 4 and e^, £^ and are extracted 
from Xj (t) before the averages are taken: 



Xj (t+1) = F(^ , {m"- 1 (t)-m /i+1 (t)}- 



JV 



E e ^ +1 ^w 



i 



Jv/(i - /) 



1 JV P 



JV/(1 - /) 



(A16) 



7 



where x 



'(aOO£) 



a?j (i + lj + Uj (i)^ 

(t), which is independent of both £^(t) and e^, is the differential of a; 



(A17) 



(t) = a , {m"- 1 (t)-m"+ 1 (f)}- 



1 



TV 



(/*)(<&: 



t 



(i) and 



4cr^ +i ^(i), 



Nf(l-f) 



i(t) 



(A18) 



F( % (t)-^'' <J (t)-0),(A19) 
^M^*) (t+1 ) = F'( % (t)- U ^ } (t)-^).(A20) 

The first, second and third term in the right-hand side 
of equation i|A18(l is order of 1/ >/N with respect to N 
and the fourth term is order of 1/N with respect to N. 

In the thermodynamic limit, N — ► oo, Jt \t) is small 
and equation 1A16I) equals equation ljA17|) . Substituting 
equation l|A17|l into the right-hand side of equation (|A15|) 
and averaging the resultant expressions over £j* and 
and using Efe^J = yield to the following equation for 
m» (t)(fi ± t): 

m"(t) 



f 



TV 



E^ 



(*) 



Nf(l - /) 



E(^) 2 {' 
-Ht-i)}^^*) 



(t-i) 



(A21) 



TV 



+£/(*) {m"" 1 ^ - 1) - m^ +1 (t - 1)} , 

(A22) 



where J7(t) = X^Li ^i^fy)- Substituting equation 



22(1 into equation (| A2|) yields 



«r -er>'(*) 

1 AT P 

+ W 7T3 7 yEE(r 1 -r 1 )^rw 

+E c/ (*){^ +lm/1_1 (*- 1 ) 

-2eT 1 m"- 1 (i - 1) + g i - 1 m" +1 (i - 1)} 

1 AT p 



Substituting equation l|A17|) into the last term of equa- 
tion (|A23(I yields the following expression for Uj(t): 



JV p 



( (t) = - g-ww + m l_ T) e E(er +1 - fr x )ffrf(*) 
+i>w {erv"- 1 ^ - 1) - 2er i ™^ i (t - 1) + cr 1 ^ 1 ^ - 1)} 



TV p 



TV p 



(r 1 - r v 4 w + j^jt^ e E(r 1 - r w w 

+ !>(*) {^'(t - 1) - 7^- x mT x {t - 1) +C"r 1 m Al+1 (t - 1)} 



TV p 



1 V V e^V^V^^^m 



#/(!-/) 



.7=1 M=l 



(A24) 
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N p 

whT)]^ E {^(t - 1) - m^(t - 1)} x 'j^\ t ) 

/ i \ 2 AT p JV 

+ ( AFTTT-n ) E E ^rer ^ E - d^^w 

3=1 M=l 
2 N p p 

+ ( /vttt^tt ) E E C^r 1 ^ E ^er 1 ^ - i^^w 

2 iV p 



AT/(1- 


/) 


1 




AT/(1- 


/) 


1 




AT/(1- 


/) 



+ (/v77T^n) EEWW 1 ^*-^ 1 ( A25 ) 

The fifth, sixth, seventh and eighth terms vanish since E[e^] = 0, and this yields 

AT p 

= - g->«(t) + jV/(1 _ T) e E^r +1 - er x )^f w 



E^W 1) - 2Cr 1 ^- 1 (t - 1) + |"f" 1 m' i+1 (i - 1)} 



AT p 



where 



-^TlEEi^r^^^'w (A26) 

- (£ +1 -rV'W + ^), (A27) 

iV p p 

Zl{t) = Nf(l-f) E E(^ +1 - er'KT^W + E U(t){g +1 m»-\t - 1) - 2^- i m^- i (t - 1) 

N p 

1)} -^EEW^'W" (A28) 

(i) is the cross-talk noise term. We assume that the cross-talk noise obeys a Gaussian distribution with mean and 
time-dependent variance <J 2 (t): E[zi(t)] = 0, E[(zi(i)) 2 ] = a 2 {t) 0,|2£]]- The first and second term of Zi(t) are the 
same cross-talk noise term as that of our previous work . The square of Zi (t) is given by 



/ | \ 2 N P 



E c/w^er 1 ™"- 1 ^ - 1) - 2^- i m"- i (< - 1) + ir'^+^t - 1)}' 

N p 2 

EE(4 r V(erV(^ 2 {^ )M w} 



i \ 2 AT 



iv/(i - /) 

2 



j'=i m=i 

TV p 

v/(l ^EE^-rWw 
j v jy j=i ^ 



x E - 1) - 2^'- 1 m'* , - 1 (< - 1) + ^'- 1 m^' +1 (t - 1)} 

p 

+2j2u{t){^ +1 m^\t - 1) - 2eT 1 m' i - 1 (i - 1) + g , - 1 m" +1 (* - 1)} 



AT p 



9 



2 ( 1 

U/(i-/) 

1 

Nf(l - /) 



2 N p , i N p 



E E < V 'ef 

2 -/V P 

EE^-cr 1 ) 2 ^) 2 ^'^)}' 
E t/w^r 1 ™^ - 1) - 2£"r 1 ^- i (t - 1) + fr 1 ™^ - 1)} 

N p 2 

EE(C 1 ) 2 (er 1 ) 2 (^) 2 {^ )(e - ) w} 



fc=i ^'^t 



JV/(1 - /) 



2 JV 



3=1 M=l 
t a 

E a(a+i)C( +i)Qtg(< ~ a) II ^ - fe + !) 

I 



aS 2 



6=1 



where p = aN, g(t) = ^E<=i (*)}'. = 
a! (b^— a.) ! ' a ' ^ s ^ c factorial with positive integer a and 
E[(ef ) 2 ] = S 2 . Since E[ef ] = 0, the fourth, fifth and 



sixth terms in equation (|A29I) vanish. We applied the re- 
lationship $3 _o(&C ) 2 = 2&Cb in this derivation. Since 
E[zj(t)] = 0, the variance cr 2 (t) is equal to E[{zi(£)} 2 ]. 
We then get the recursive equation for cr 2 (t): 



° 2 {t) = ^2( a +i)C (a+1) aq(t-a)Y[U 2 (t-b+l) 



6=1 



aS 2 



(A32) 



The overlap between the state x(t) and the retrieval pat- 
tern £ is given by 



m'(t) 



i N 

Nf(l ~ f) JJ (e ' " /} 



((tf-g-V'-^t-ij + ^t-i)-*). 

(A33) 



Since Ui(t) is independent and identical distribution 
(i.i.d.), by the law of large numbers, the average over i 
can be replaced by an average over the memory patterns 
£ p and the Gaussian noise term z ~ 7V(0, cr 2 ). Then, the 
recursive equation for the overlap m'(i) is transformed: 



i\t) 



1 1 



dze 



f) 



/(I - /) yptoo 

i i 

/(l-/)v/2^ 



:«(*)» 



(A30) 
(A31) 



(A34) 



xF 



= ^^-erf(0 o ) - ^erf(0i) + {erf(0 2 ), (A35) 
where z — z/a, erf(y) = ^= JjJ* exp (— u 2 )du, and 0o = 

9 i -m t - 1 (t-l)+9 , _ m t_1 (i-l)+0 , 

V2a(t-1) ' VI - ^(t-1) ' ^ 2 - V2*(t-1) ' and 

((•)) denotes an average over the memory pattern 
Since a^(t) — x'^ it) ~ ^(r^y) and the thermodynamic 

limit, A~ — ■> co, is considered, x'^(t) — x'^t). Using this 
relationship, we derive U(t): 



N N 

u ® - ^E^w = ^E^) 



(A36) 



i=i 
1 1 



dze 2 



x((f'((C*-e t - 2 )m t - 1 (t-l) 



(A37) 



1 1 



<ize 2 z 



/(i-/)V2¥7_ c 

x((^((e t -^- 2 )m t - 1 (t-l) 

+<r(t-l)«-fl)^ 

((1-2/ + 2.f 2 )e- 
^/2^^a(^-l) I V ; 

+/(l-/)(e- 0? +e-* 2 )}. 



(A38) 



(A39) 



Since ar,(t) - asj" ~ 0(-^=) and AT — >• oo, 
cci(t) . Using this relationship, we derive g(t): 



JV JV 

«(*) = iEK w (*)} 2 4D^*)} 



(A40) 



i=l 



10 



/(I - /) 



dze 



x((^((^-r 2 )m*- 1 (i-l) 



= l{l - (1 - 2/ + 2/ 2 )crf(^ ) - /(l - /)(erf(0 1 ) 
+erf(&))}. (A42) 



(A41) 
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